Phoneme - Viseme Mapping for German Video - Realistic Audio - Visual - Speech - Synthesis IKP - Working Paper NF 11

In this working paper we introduce a German viseme set which we already use in our data-driven audio-visual synthesis system. The viseme set is essential for speech driven audio-visual synthesis due to the fact that the selection of appropriate video segments is based on the visemically transcribed input text. For text-to-speech synthesis, a transcription of the input text into the phonemic representation is used, in order to avoid ambiguous meanings and to acquire the correct pronunciation of the underlying input text. The transcription also serves as label in unitselection based synthesis systems. Likewise, the visual synthesis requires a transcription that represents analogue to the phonemes, the visual counterpart which is called visemes in related literature and serves also as unit label in our data-driven video-realistic synthesis system. A viseme could include several phonemes, since we can not for example visually differentiate between pronouncing a /b/ or a /p/. Thus, both sounds are bilabial plosives and consequently, have the same lip-based realisation. Therefore, we worked out an inventory of German viseme classes and developed a phoneme-viseme mapping for German that we currently use in our data-driven exemplary audio-visual synthesis system. The visemes are represented in a SAMPA-like labelling. For transcribing a large corpus, we trained a maximum entropy model for automatic visemic transcription and thus received feasible results.